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Executive Summary 


W ith the spread of adaptive technologies that customize the 
user experience in response to individual users, it is not 
surprising that such experiences are increasingly found in 
educational settings or in tools to facilitate learning. The National 
Academy of Education commissioned a background paper and held 
two meetings of scholars, policy makers, and developers of adaptive 
educational technologies to consider the implications of such adap- 
tive systems for education research and education researchers. This 
report highlights the issues discussed in those proceedings. 

Recent progress in the development of adaptive educational 
technologies builds on several decades of efforts to use computer 
systems to offer tailored instructional experiences to students. Adap- 
tive elements can be found in many forms and formats that support 
learning. Adaptive hypermedia learning systems, intelligent tutor- 
ing systems, adaptive elements embedded into online courses, and 
a variety of educational games and simulations can all be designed 
to tailor the learning experience to the needs of individual students. 
A wide array of information on learners can be used to shape the 
individual learning experience, including prior achievements, pref- 
erences, interests, traits, and the immediate learning environment. 
Whatever the individual learner characteristics or the dimensions of 
the learning experience represented in the system, the key defining 
feature of adaptive educational technologies is that one or more ele- 
ments of the system are modified in response to information about 


1 


2 


ADAPTIVE EDUCATIONAL TECHNOLOGIES 


the learner. It is this adaptivity that creates the personalized learning 
experience intended to maximize the learning of each student. 

The growth of adaptive educational technologies offers some 
unique advantages as well as some new challenges for education 
research. Among the advantages are the ability to gather expanded 
learner profile metadata and aggregate it to allow us to learn more 
about the patterns of learning across venues, expanded data sources 
to make inferences about learning, and new techniques such as data 
mining and machine learning for making sense of information on 
learning. In addition, adaptive educational technologies can sup- 
port learner agency by extending access to learner data systems to 
learners themselves, they can allow us to inquire into the impact of 
learner feedback in social systems by granting learners access to data 
on their own performance in relation to the performance of others, 
and they can provide large-scale test beds for experimentation. 

In addition to new advantages, adaptive educational technolo- 
gies also present some new challenges. Adaptive systems typically 
do not include contextual data beyond the system, they gather data 
in ways that are not easily organized for research and analysis, 
and they often produce data for which it is challenging to attribute 
meaning in the absence of a prior learning theory. In addition, there 
are unresolved concerns about the ownership of data on students as 
well as concerns about student privacy. 

Although the era of adaptive educational technologies is just 
dawning, there are already several major areas of research. A num- 
ber of studies have attempted to provide effective feedback to stu- 
dents as they utilize adaptive systems. Such studies have explored 
the possibility of providing advice to students in real time using 
ideas drawn from areas such as machine learning and social network 
analysis. A second group of studies has attempted to determine how 
to provide feedback to instructors as they utilize adaptive systems, 
including information on how to improve course structure, teaching 
styles, and student motivation. A third group of studies uses data 
from adaptive systems to provide insights into patterns of student 
learning in response to particular configurations of content. These 
studies of how students learn specific subject matter offer guidance 
for the development of efficient learning trajectories. A fourth group 
of studies draws on data from adaptive educational technologies to 
inform improvements to those same systems. A fifth and final set of 
studies uses data from adaptive systems to advance general theory. 

Because adaptive educational technologies represent a substan- 
tially new research opportunity, making full use of them will require 
new types of infrastructure. Adaptive educational technologies pres- 
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ent new kinds of data organized in fresh ways. These data exist in 
new configurations that vary from system to system. Moreover, 
the conditions under which data are gathered in these systems dif- 
fer from those typically associated with education research activi- 
ties. Dealing with the situations presented in adaptive educational 
technologies will require consideration of at least new training 
for researchers, development of new tools of analysis, specifica- 
tion of prototypes and standard protocols, and attention to the rights 
of individuals whose data are collected as part of the operation of 
these systems. 

Education researchers will require new skills to make use of the 
data generated by adaptive technologies. Elements of an effective 
training infrastructure might include seminars and workshops on 
handling large and complex datasets; publications such as hand- 
books, textbooks, and journals to build the knowledge base on tech- 
niques for dealing with data from adaptive systems; specialized 
professional associations and related conferences; and courses, spe- 
cializations, and degree programs in institutions that prepare educa- 
tion researchers. Developing prototypes and protocols to standard- 
ize the data produced by adaptive educational technologies could 
also support the greater use of such data by researchers. In addition, 
investments in the development of analytical tools to handle data 
from adaptive educational technologies could reduce the burden on 
education researchers and encourage greater use of data from such 
systems. Finally, developing models for the governance of the data 
generated by adaptive systems will be necessary to promote access 
for education researchers. 

In view of the rapid growth of the use of adaptive educational 
technologies, workshop participants identified some possible next 
steps to protect the interests of the education research community. 
These include developing standards for data gathered through 
adaptive educational technologies to support education research, 
developing standards for credentials for education researchers to 
demonstrate proficiency in the handling of data from adaptive edu- 
cational technologies, and guidelines for human subjects commit- 
tees to facilitate the review of research projects involving data from 
adaptive systems. 

The growth of adaptive educational technologies presents new 
opportunities for education research that can advance our under- 
standing of student learning and performance. The full participation 
of the education research community is necessary to create the con- 
ditions that will guarantee that the promise of adaptive educational 
technologies is fully realized for research as well as practice. 
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Introduction 


W e have reached a point where most people in modem 
societies have at least some experience with adaptive 
technologies, that is, systems that present themselves in 
ways customized to individual characteristics. Such systems are all 
around us: the lock that opens in response to your ID card, the ATM 
that retrieves your account information, the car seat that returns to 
the position you left it in when you insert your key (despite what 
other drivers may have done in the meantime), the thumbprint 
reader that allows only you to access your computer. 

In the online world, the examples are even more powerful: per- 
sonalized pages on Amazon that show items that may be of interest 
to you given your browsing and purchasing history, Google search 
results personalized in response to your search history, the personal 
account information displayed by your bank after you log in. 

The behavior of these and similar systems is modified or adapted 
in response to individual users. The systems may respond to direc- 
tions or information provided by the user, to an action or choice, or 
to information in the system or in connected systems. 

With the spread of adaptive technologies into all aspects of life, 
it is not surprising that they are increasingly found in educational 
settings or as tools to support learners and to facilitate learning. The 
growth of such technologies has implications for educators, learners, 
and all those interested in using them in tools for learning. 

The growing use of adaptive educational technologies as impor- 
tant elements in the education sector also creates new opportunities 
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and challenges for education researchers. These tools generate new 
types of robust datasets that can offer new possibilities for education 
researchers. At the same time, these new opportunities suggest the 
need to enhance the skills of education researchers so that they can 
manage data from adaptive systems and utilize the data in a range 
of basic and applied studies. 

Over the course of 2011, the National Academy of Education 
commissioned a background paper and convened two meetings 
to discuss these and related issues. An initial planning meeting 
was held in April to identify major issues and topics for a more 
extensive gathering in December. The December meeting included 
panels on learning, instruction, assessment, concerns, institutional 
responses and innovations, and developing infrastructure as well 
as demonstrations of a number of adaptive educational systems. 
Appendixes A and B contain the lists of attendees and panelists 
from the meetings. This report highlights the issues discussed in 
those proceedings. 
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What Are 

Adaptive Educational Technologies? 


T he goal of responding to the needs of individual learners has 
received attention recently because of new demands on the 
educational system and new possibilities for providing per- 
sonalized learning support. New demands for personalized learning 
stem from the growing sense that advanced economies require the 
vast majority of citizens to achieve high levels of learning through- 
out their lives. Such a massive increase in the demand for education 
may only be met with new tools, techniques, and learning resources. 

Progress over the past decades in computing and communica- 
tions technologies has set the foundation for a new learning infra- 
structure (Computer Research Association, 2005; National Science 
Foundation Task Force on Cyber learning, 2008). Many computer- 
based systems engage students with educational opportunities, 
including instruction and resources. There has also been a shift from 
stand-alone hypermedia and tutoring systems to widely available 
Web-based systems. It is upon these technologies that new learning 
tools are being built to support individually responsive learning 
environments (Gardner, 2009; Maeroff, 2003) that promise to help 
greater proportions of the population to achieve higher levels of 
learning. 

Adaptive educational technologies take account of current 
learner performance and adapt accordingly to support and maxi- 
mize learning. By design, they present personalized educational 
experiences for each learner. Such technologies grow out of a long 
line of work on using computer systems to offer tailored instruc- 
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tional experiences to students (U.S. Congress, Office of Technology 
Assessment, 1988), beginning with Skinner's teaching machines in 
the 1950s (Skinner, 1986), and continuing on through the PLATO 
project at the University of Illinois in the 1960s (Smith & Sher- 
wood, 1976), and the work of Suppes and Atkinson at Stanford on 
computer-assisted instruction in the 1970s and beyond (Suppes & 
Fortune, 1985). 

Adaptive elements can be found in many forms and formats that 
support learning. They might involve ways of organizing resources 
or might involve complex learning environments. Adaptive hyper- 
media learning systems organize and present resources in ways that 
are tailored to individual student learning needs (Brusilovsky, 2001). 
Intelligent tutoring systems attempt to achieve the kinds of posi- 
tive impact on learning long associated with one-on-one tutoring 
(VanLehn, 2011). The instructional elements of these systems, which 
are based on domain knowledge, knowledge of typical student 
learning patterns, knowledge of teaching strategies, and knowledge 
of methods for communicating with students (Woolf, 2009), adapt in 
response to individual students in order to maximize learning. Tutor- 
ing and other adaptive strategies can be embedded in online courses 
(Lovett, Meyer, & Thille, 2008). Additionally, a small but growing 
number of educational games utilize adaptive techniques to enhance 
the learning of individual players (Pierce, Conlan, & Wade, 2008; 
Barab, Gresalfi, & Ingram-Goble, 2010; National Research Council, 
2011; Reese, 2012; Shute & Ventura, 2013). 

A wide array of information about learners has been used to 
drive adaptation, including the learner's current state of knowledge 
and history of learning as inferred from his or her digital educational 
system interactions. Of course, both of these are considered within 
the context of the learning goals established within a particular 
system. More general characteristics of the learner — preferences, 
interests, and traits — are also used in some adaptive systems. The 
learner's experience with online environments and the immediate 
environment in which the learner is working may also be consid- 
ered. Information on all of these factors can be used to generate the 
optimal personal learning experience through the adaptive system 
(Brusilovsky, 1996, 2001). 

Brusilovsky (1996) specifies two broad techniques for adapting 
content to learners: adaptive presentation and adaptive navigation. 
Adaptive presentation involves tailoring the presentation of media 
content, presenting different text to different learners, and adapta- 
tion of the mode of presentation. Adaptive navigation involves tech- 
niques that help learners navigate content by adapting the way links 
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are presented. Adaptive navigation techniques include providing 
direct guidance to learners, sorting links, hiding links, annotating 
links, generating links, and mapping links, all based on individual 
learner characteristics (Brusilovsky, 2001). 

In terms of the more complex learning arrangements of virtual 
worlds and games, the possibilities for adaptation become greater 
and less generic. In the panel on instruction, Sasha Barab talked 
about "adapting whole story lines, whole worlds, whole roles, not 
just conceptual ideas. . . Indeed, such complex learning environ- 
ments allow elements such as role specifications and entire story 
lines to respond to individual learners and their unique characteris- 
tics (Barab, Gresalfi, & Ingram-Goble, 2010). 

Whatever the individual learner characteristics or the dimen- 
sions of the learning experience represented in the system, the key 
defining feature of adaptive educational technologies is that one or 
more elements of the system are modified in response to informa- 
tion about the learner. It is this adaptivity that creates the personal- 
ized learning experience intended to maximize the learning of each 
student. 
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The Potential of Adaptive 
Educational Technologies in 
Education Research 


B ecause adaptive educational technologies collect data on indi- 
vidual students and student performance, they generate data- 
sets that offer both new opportunities and new challenges for 
education researchers. 


Unique Advantages of Using Data from 
Adaptive Learning Technologies 

In the panel on learning, Roy Pea identified six ways in which 
adaptive learning technologies can help us learn about learning: 

1. Adaptive learning technologies provide expanded learner 
profile metadata and aggregate them to allow us to capture 
the benefits at scale of learning more about the patterns of 
learning across diverse schools, districts, and states. 

2. They expand the data sources we use to make inferences 
about learning and its conditions. 

3. Through the use of techniques such as data mining and 
machine learning, we can expand our sense-making tech- 
niques for understanding learning and related conditions 
as a basis for guiding more effective learning. 

4. Adaptive learning technologies can extend access to learner 
data systems to the learners themselves to enhance agency, 
self-assessment, and self-regulation. 
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5. They can also extend learner access to data about their own 
performance in relation to the performance of others to 
support inquiry into the functioning of learner feedback in 
social systems. 

6. Adaptive educational technologies can also provide large- 
scale test beds for experimentation. 

These new opportunities afforded by adaptive educational tech- 
nologies suggest three courses for education research. First, they 
offer new possibilities for education researchers to examine prob- 
lems and issues previously examined in other data. Second, they 
allow new kinds of research questions that have previously eluded 
empirical examination. Third, they have the potential to generate 
research questions as a result of examinations of large new datasets. 
However, as is discussed below, the data generated by adaptive 
systems come with special challenges as well. 


Unique Challenges of Using Data from 
Adaptive Learning Technologies 

The data generated by adaptive educational technologies pres- 
ent challenges for analysts trying to extract meaning and address 
research questions, and also present some special issues worth 
noting: 

1 . Although data gathered through adaptive systems can offer 
insight into important relationships among the variables 
included, they typically do not include contextual data be- 
yond the system itself. Thus, for example, data on events 
preceding student engagement with the system, or data 
on contemporaneous events outside the system such as 
conversations between students and teachers or learning 
experiences in the home or community, would not be avail- 
able without additional efforts to collect data. In addition 
to such traditional out-of-system events, during the meet- 
ing of the panel on learning, Jim Gee highlighted the inde- 
pendent growth of affinity spaces where users of systems 
gather to share knowledge; activity in such spaces would 
not be captured in the data generated by the adaptive sys- 
tem. If the adaptive system generates data over a consider- 
able period of time, the lack of data on other aspects of the 
students' educational and life experience may compromise 
a researcher's capacity to develop a clear understanding 
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of the impact of experience on learning. Thus, researchers 
relying solely on data from the adaptive system might not 
appreciate the contextual factors. 

2. Adaptive technologies gather data in ways that are opti- 
mized for the efficient operation of the system, including ad- 
justment of what the system presents to students. Such data 
are not organized in ways that are amenable for analysis. At 
the meeting, Brian Rowan described the challenge of taking 
data from adaptive systems and processing them to make 
them suitable for analysis. Preparing data from adaptive 
systems for analysis is a very substantial task. As a result, 
it may be more accurate to view the data preparation stage 
as another step of data collection as the researcher selects 
and reorganizes the data to address the research questions 
(Romero & Ventura, 2007). 

3. While adaptive educational technologies tend to produce 
data tightly linked to specific student actions, the mean- 
ing of such data is often unclear. For example, the kind of 
keystroke data typically gathered may raise questions about 
the proper unit of data for analysis. Less detailed and more 
meaningful actions are both easier to handle (Stephens & 
Sukumar, 2006) and potentially more useful for research 
purposes (Mislevy et al., 2010). Other student movements 
(e.g., drawing on a tablet or making gestures) are challeng- 
ing to interpret, and attributing meaning requires additional 
data, assumptions, or, ideally, a framework of meaning. 

4. Although there are ongoing efforts to address issues of data 
ownership and access (Office of Science and Technology 
Policy, 2012), the ownership of and access to data from 
adaptive educational systems is complicated. Education 
researchers are accustomed to negotiating access with 
students, parents, and schools, but adaptive educational 
systems introduce another player into the mix: the system 
developer or provider. System providers may assert rights 
to the use of data for system development, and they may 
be reluctant to share proprietary data rights with education 
researchers. Of course, commercial providers are not unique 
in that regard as John Stemper suggested in the panel on 
concerns when he explained the challenge of encouraging 
researchers to share data in open repositories. 

5. Because adaptive educational technologies gather informa- 
tion on individual students through online applications, they 
inherently raise three types of privacy concerns. First is the 
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set of concerns related to the status of students as individual 
citizens or consumers in a networked world where valuable 
access to networked resources requires the exchange of per- 
sonally identifiable information (Nissenbaum, 2010). Second 
is the set of concerns related to the status of many students 
as children whose privacy may require additional protec- 
tions by virtue of their youth (Pitman & McLaughlin, 2000). 
Third is the set of concerns related to the role of students 
within educational institutions, a role that entails the gath- 
ering of particular kinds of information in the educational 
process (Glenn, 2008). Developers, providers, and adopters 
of adaptive educational technologies must confront these 
multiple layers of concerns for the privacy of student users 
of such technologies. Education researchers intent on using 
the data generated by adaptive technologies must be ac- 
countable for understanding these various privacy concerns 
and the procedures for addressing them in the applications 
that generate data used in their research. 

6. Because data gathering in adaptive systems is integrated 
with program delivery in a way seldom encountered in edu- 
cation research activities that are typically grafted on (often 
over considerable resistance) to the regular business of edu- 
cational programs, the opportunities for research have the 
potential to expand exponentially. 

The unique opportunities afforded education researchers by 
data from adaptive educational technologies have been and will 
be sufficient to generate the interest and effort necessary to address 
the unique challenges presented by their use in research. Indeed, 
some of the challenges noted can be reduced or eliminated over 
time. For example, as learning scientists become more involved in 
the design and development of adaptive learning systems, these 
systems are likely to be designed with research and data interpreta- 
tion in mind. As adaptive systems become more widely used, issues 
of data ownership, handling, and privacy are likely to be resolved. 

Even at this early stage, adaptive educational technologies are 
supporting fruitful lines of inquiry. In the next section are some 
examples of research drawing on data from adaptive systems. 
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Examples of Research Drawing on 
Adaptive Educational Technologies 


S everal attempts to characterize and classify research that makes 
use of data from adaptive educational technologies (Baker & 
Yacef, 2009; Castro, Vellido, Nebot, & Mugica, 2007; Romero 
& Ventura, 2007) highlight the breadth of possibilities. To illus- 
trate the types and range of studies, five categories are highlighted 
below with the caveat that some studies may fall into more than 
one category. 


Major Areas of Research on 
Adaptive Educational Technologies 

• Research that informs student users of adaptive systems 

• Research that informs teacher users of adaptive systems 

• Research that informs curriculum development 

• Research that informs the design and improvement of adaptive systems 

• Research that advances general theory and practice 


Research That Informs Student Users of Adaptive Systems 

A number of studies have attempted to determine how to pro- 
vide effective feedback to students as they use adaptive systems. 
These studies have explored the possibility of providing advice to 
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students in real time using ideas drawn from, among other areas, 
machine learning and social network analysis. Examples of this type 
include the following: 

Hwang (1999) investigated a system to provide learning 
advice to students. 

Heraud, France, and Mille (2004) used student log data to 
guide students in a tutoring system. 

Kelly and Tangney (2005) employed machine learning tech- 
niques to generate information on student learning styles. 
Romero, Ventura, Zafra, and De Bra (2009); and Tang and 
McCalla (2005) provided personalized content for students, 
the former via Web-usage mining, and the latter through 
social network analysis. 


Research That Informs Teacher Users of Adaptive Systems 

Studies in a second group have attempted to determine how to 
effectively provide feedback to instructors as they utilize adaptive 
systems, including information on how to improve course struc- 
ture, teaching styles, and student motivation. Examples of this type 
include the following: 

Feng and Heffeman (2007) provided live reporting to teachers 
on student performance in the ASSISTment System. 

Romero, Venturo, and De Bra (2004); Tang, Lau, Li, Yin, Li, 
and Kilis (2000); and Vialardi, Bravo, and Ortigosa (2008) 
provided general insights for course development and 
improvement. 

Roll, Aleven, McLaren, and Koedinger (2011) drew on stu- 
dents' help-seeking to infer learning. 

Hurley and Weibelzahl (2007) provided insight into student 
motivation. 

Crespo, Pardo, Perez, and Kloos (2005); and Zakrzewska 
(2008) developed information to guide group formation. 


Research That Informs Curriculum Development 

Some studies using data from adaptive systems provide insights 
into patterns of student learning in response to particular configura- 
tions of content. These studies of how students learn specific sub- 
ject matter offer guidance for the development of efficient learning 
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trajectories. Examples of this area of research discussed at the meet- 
ing include the following: 

Baker (2007), Jong, Chan, and Wu (2007), and Muehlenbrock 
(2005) worked to detect student responses to the learning of 
specific content areas. 

Simko and Bielikova (2009) developed concept maps of spe- 
cific subject-matter areas with the goal of allowing instruc- 
tors to automatically create graphs showing the relationships 
among concepts and the hierarchical nature of knowledge in 
those domains. 

Pavlik, Cen, and Koedinger (2009) analyzed learning curves 
to generate domain models. 


Research That Informs the Design and 
Improvement of Adaptive Systems 

Other studies have attempted to draw on data from adap- 
tive educational technologies to inform how improvements to the 
design of these systems might be made most effectively. For exam- 
ple, studies have attempted to understand how different types of 
students respond to various adaptive educational technologies. 
Others have focused on the delivery models for different forms of 
content or across different subject matter, and still others on the 
efficacy of various pedagogical strategies. Examples of this area of 
research include the following: 

Chi, VanLehn, Litman, and Jordan (2010) examined peda- 
gogical approaches that lead to effective tutoring experiences. 
Superby, Vandamme, and Meskens (2006) identified factors 
that predict student failure in college. 


Research That Advances General Theory and Practice 

A fifth and final type of study has attempted to use data from 
adaptive systems to advance general theory. Self-regulation is one 
area where general theory has been used in research on adap- 
tive systems (Lajoie & Azevedo, 2006). Research on the impact of 
hypermedia environments on self-regulated learning provides a 
good example of work that uses data from adaptive systems to 
refine understanding of theory and ultimately improve practice 
beyond the immediate adaptive system. Specific examples include 
the following: 
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• Azevedo, Guthrie, and Seibert (2004) examined student use of 
self-regulated learning processes and the impact on learning. 

• McManus (2000) analyzed the relationship between levels 
of learner control in hypermedia environments and student 
self-regulatory skills for their impact on learning. 
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Infrastructure Needed to 
Support Research Drawing on 
Adaptive Educational Technologies 


B ecause adaptive educational technologies represent a sub- 
stantially new research opportunity, making full use of them 
will require new types of infrastructure. These technologies 
present new kinds of data organized in fresh ways. These data 
exist in new configurations and in ways that vary from system to 
system. Moreover, the conditions under which data are gathered in 
these systems differ from those typically associated with education 
research activities. Dealing with the situations presented in adap- 
tive educational technologies will require consideration of at least 
four responses: training for researchers, development of new tools 
of analysis, specification of prototypes and standard protocols, and 
attention to the rights of individuals whose data are collected as part 
of the operation of these systems. 


Training for Researchers 

Education researchers will require new skills to make use of 
the data generated by adaptive technologies. Such skills are neces- 
sary for the tasks that are involved in taking the data from adap- 
tive systems and making them manageable in analyses. Education 
researchers will also require skills in analyses that are more com- 
mon in data mining typically conducted by computer scientists and 
systems engineers (Romero, Ventura, Pechenisky, & Baker, 2011). 
Moreover, if researchers wish to participate in the development of 
adaptive systems and thereby have a say in the data that are col- 
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lected, they will require knowledge of the designs and the design 
possibilities of the systems. Otherwise, researchers will be limited to 
data available from systems designed without their research ques- 
tions in mind. 

With a relatively well-defined set of skills to be conveyed to 
education researchers, several elements of what might become an 
effective training infrastructure are essential: 

1. Seminars and workshops devoted to handling the large and 
complex datasets generated by adaptive learning technolo- 
gies could provide focused training and practice oppor- 
tunities. A model for such activities can be found in the 
institutes and workshops sponsored by the National Center 
for Education Statistics to prepare researchers to work with 
national datasets. Another option would be to organize 
such activities through a network of regional institutionally 
based programs. Examples of this approach can be found in 
the Pittsburgh Science of Learning Center's summer school 
on mining of data from adaptive learning technologies and 
the Learning Analytics Summer Institutes at Stanford. 

2. A variety of publications might contribute to a knowledge 
base on techniques for dealing with data from adaptive sys- 
tems. These might include publications such as the Handbook 
of Educational Data Mining (Romero, Ventura, Pechenisky, 
& Baker, 2011) as well as textbooks and other professional 
books highlighting the evolving set of analysis issues 
and techniques. Specialized journals, such as the recently 
launched Journal of Educational Data Mining and the Journal 
of Learning Analytics, could offer outlets for publication and 
modes of studies using the new kinds of datasets. 

3. Specialized professional associations might be formed to 
create opportunities for education researchers to learn 
from one another via conferences and other activities. 
Education researchers could also join existing communi- 
ties of researchers in intelligent tutoring systems, artificial 
intelligence in education, educational data mining, learning 
analytics and knowledge, and the International Society of 
Learning Sciences. 

4. Perhaps the most substantial element of a new training 
infrastructure would be the incorporation of courses, spe- 
cializations, and degree programs in the graduate schools 
where education researchers are prepared and in collab- 
orative efforts to create joint programs with departments 
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of computer science, statistics, psychology, and sociology 
where big data science is being developed and where edu- 
cation research topics are coming to be addressed. This, of 
course, would require the preparation of faculty to handle 
such efforts. 


Prototypes and Protocols 

Providing training to education researchers to take on the cur- 
rently nonstandard and unwieldy datasets emanating from adap- 
tive learning technologies is only one approach to developing an 
infrastructure to support education research on adaptive systems. 
Another approach involves the development of prototypes and 
protocols that could lead to greater standardization of the data pro- 
duced by adaptive educational technologies. Education researchers 
have a role in developing prototype systems, particularly around 
data gathering and reporting. In addition, organizations such as the 
U.S. Department of Education with its Learning Resource Metadata 
Initiative and Learning Registry and the Schools Interoperability 
Framework Association with its SIF protocol offer models of the 
kind of standard setting that may alleviate some of the difficulties of 
accessing data from diverse systems from any number of providers. 
Investments in the development of publicly shareable prototypes 
and protocols could accelerate the use of data from adaptive tech- 
nologies in education research. 


Tools 

The tools used for the analysis of data from adaptive learning 
technologies have not been developed with education researchers 
in mind. Most of the tools are generic and have a steep learning 
curve for scholars outside the specialized areas of inquiry, often 
in computer science, for which they have been developed. This 
means that education researchers new to adaptive technology will 
need to invest considerable time and resources to make good use of 
the available tools. Investments in the development of tools would 
reduce the burden on researchers and encourage use of data from 
adaptive technologies for education research. These investments 
could take the form of improved documentation, more intuitive and 
understandable interfaces, and enhanced technical support. 
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Issues in Creating a Data Governance Model 

Education researchers are accustomed to dealing with the com- 
plexities of securing access to data about students and their learning. 
Such efforts address issues of informed consent and the protec- 
tion of human subjects. They meet the requirements of human sub- 
jects committees at the institutions where the research is based as 
well as requirements of the schools and districts where the data are 
gathered. 

The data gathered via adaptive educational technologies present 
new complexities for all concerned. Adaptive systems capture data 
on students and their learning in ways that may not be transpar- 
ent to either students or their parents. Because adaptive systems 
are often operated by software vendors, publishers, or other third 
parties, and because the data are often located in systems physi- 
cally outside the schools and districts where they are collected, the 
various rights to the data may not be clear. This, coupled with the 
standards common for other education research (e.g., informed con- 
sent), currently presents barriers that make it difficult for researchers 
to use data from adaptive systems for education research. However, 
there are efforts under way to overcome such barriers through the 
development of principles, policies, and practices to leverage the 
value of individual data while protecting the privacy rights of indi- 
viduals. Such efforts include those of the U.S. Department of Educa- 
tion (2012), the OECD (2012), and the World Economic Forum (2012). 
Nevertheless, challenges to the evolving policies regarding student 
data suggest that issues involving student data are far from settled 
(Electronic Privacy Information Center, 2013). 

Addressing the complexities of data rights will require the devel- 
opment of a management governance process to specify the vari- 
ous rights to data. Elements of the management governance must 
include: 

1. A clear understanding of the kinds of data gathered by 
adaptive educational technologies; 

2. Specification of the various rights that may be associated 
with data (e.g., the right to delete or modify data, and the 
right to use data to enhance the educational experience, 
for system improvement, and to address general research 
questions); 

3. The parties who have an interest in the data on students 
and their learning (e.g., students, parents, teachers, schools, 
districts, states, system providers, colleges, employers, gov- 
ernance organizations); 
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4. The relationships among the various parties; 

5. The conditions under which specific rights may be exer- 
cised; and 

6. The precautions required to protect the interests of each 
party involved. 

Developing one or more models for a data rights governance 
process will save considerable time and expense for all concerned 
as they develop data-sharing arrangements. 


Additional Reading 

For more on the topic of data mining and adaptive learning tech- 
nologies, see Enhancing Teaching and Learning Through Educational 
Data Mining and Learning Analytics: An Issue Brief, published by the 
U.S. Department of Education. Copies are available at: http://www.ed.gov/ 
edblogs/technology/files/2012/03/edm-la-brief.pdf. 
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Next Steps 


T he rapid growth of the utilization of adaptive learning tech- 
nologies has implications for all concerned: students, parents, 
educators, educational agencies, system developers and pro- 
viders, and education researchers. Workshop participants suggested 
some possible next steps to protect the interests of the education 
research community and the opportunities for and integrity of the 
education research process. 


Step 1. Standards for Research Data 

The education research community could develop standards for 
data gathered through adaptive educational technologies to support 
education research. Ideally, these standards would be developed 
by a consortium of research associations. These standards could be 
used to encourage developers to make provisions for gathering data 
as part of the design and development process. Additionally, the 
research community could offer a review procedure leading to the 
designation of an adaptive technology system as meeting research 
standards. 


Step 2. Credentials for Education Researchers 

The education research community could develop standards for 
a program of study leading to proficiency in using data from adap- 
tive educational technologies. Such standards might be developed 
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by a group of graduate programs in conjunction with one or more 
research associations. The completion of the program of study or 
individual components of study could result in a certificate or other 
credential. 


Step 3. Guidelines for Human Subjects Committees 

The education research community could develop guidelines 
for human subjects committees to facilitate the review of research 
proposals that involve data from adaptive educational technologies. 
These guidelines might address the major concerns posed by the 
more complicated data-gathering processes, more elaborate data 
structures, and more distributed ownership patterns associated 
with adaptive systems. The guidelines could be issued by a con- 
sortium of research associations, government agencies, and gradu- 
ate programs in education research. At the closing session of the 
meeting. Bob Hauser noted that the federal government had issued 
notice of proposed rulemaking in the area of human subjects in July 
2011 and that a key response document had been prepared under 
the leadership of Felice Levine of American Educational Research 
Association representing the work of a collectivity of social science 
groups and organizations. The National Academies' Committee on 
Revisions to the Common Rule for the Protection of Human Sub- 
jects in Research in the Behavioral and Social Sciences is currently 
working in this area with a report expected later in 2013 (see http: / / 
www8.nationalacademies.org/cp / projectview.aspx?key=49500 for 
a description of the project). 

The growth of adaptive educational technologies presents new 
opportunities for education research that can advance our under- 
standing of student learning and performance. The full participation 
of the education research community is necessary to create the con- 
ditions that will guarantee that the promise of adaptive educational 
technologies is fully realized for research as well as practice. 
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8:00-8:30 am 
8:30-8:40 am 

8:40-9:00 am 
9:00-10:00 am 


Workshop Agenda 

Planning Meeting 
May 12, 2011 
Keck Center — Room 101 
500 Fifth Street, NW 
Washington, DC 20001 


MEETING AGENDA 

Thursday, May 12 
Breakfast 

Welcome and Meeting Overview 

Susan Fuhrman, Teachers College, Columbia 
University; President, National Academy of 
Education 

Participant Introductions 

Product Demonstrations — 

Envisioning the Scope of AETs 

Chair: Susan Fuhrman 

15-minute demos — -focused equally on the product 
and on the data captured 

• MasteringPhysics (Rasil Warnakulasooriya) 

• WISE (Marcia Linn) 

• ASSISTments (Neil Heffeman) 
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10:00-10:30 am 


10:30-11:00 am 


11:00 am-12:00 pm 


12:00-12:30 pm 


12:30-3:30 pm 


Discussion — Analysis of Data from AETs 

• 10-minute presentation on the literature 
(Ken Koedinger) 

• 20-minute group discussion 

Student Learning from AETs 

• 10-minute presentation on data archiving 
(John Stamper, PSLC Datashop) 

• 20-minute group discussion 

Roundtable — Developing Models Using 
Data from AETs 

10-minute presentations, followed by group 
discussion 

• Student Affect (Bob Dolan) 

• Social Networks (Shane Dawson) 

• Teacher Implementation (Brian Rowan) 

• Intervention Effectiveness (Guido Gatti) 

Working Lunch: Discussion of Background 
Paper 

Chair: James Gee, Arizona State University 

Addressing the 3Q's and Identifying Topics 
for the Summit 

Chair: Brian Rowan, University of Michigan 

Discussion of Question #1: 

• What research opportunities are possible 
using these data? 

Discussion of Question #2: 

• What kinds of analyses have researchers 
conducted in the past using such data? 

And, what has been learned from such 
analyses? 

Discussion of Question #3: 

• What more is needed to develop research in 
this area? 

— What are the costs and benefits of using 
such data for research? 
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3:30-4:00 pm 
4:00 pm 


— What kind of organizational supports 
would be needed from developers if 
data were used for research and program 
improvement? 

— What other accommodations might be 
needed for researchers (e.g., to ensure 
confidentiality of data, allow data to be 
processed statistically, etc.)? 

Wrap-up, concluding comments, and next 
steps 

Meeting adjourned 
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WORKSHOP PARTICIPANT LIST 

May 12, 2011, Planning Meeting 


Co-Chairs: 

Susan Fuhrman, Teachers College, Columbia University 
James Gee, Arizona State University 
Brian Rowan, University of Michigan 


Participants: 

Judie Ahn, National Academy of Education 
Roger Azevedo, McGill University 
Sasha Barab, Indiana University 
John Behrens, CISCO 
Larry Berger, Wireless Generation 

Christopher Brown, Pearson Foundation Research Program 

Allan Collins, Northwestern University 

Katie Conway, Teachers College, Columbia University 

Shane Dawson, University of British Columbia 

Andrea diSessa, University of California, Berkeley 

Bob Dolan, Assessment & Information, Pearson 

Guido Gatti, Gatti Evaluation, Inc. 

Richard Halverson, University of Wisconsin-Madison 

Michael Hansen, Urban Institute and CALDER 

Aaron Harnly, Wireless Generation 

Neil Heffernan, Worcester Polytechnic Institute 

Paul Horwitz, Concord Consortium 

Caitlin Kelleher, Washington University in St. Louis 

Ken Koedinger, Carnegie Mellon University 

Carol Lee, Northwestern University 

Marcia Linn, University of California, Berkeley 

Robert Mislevy, University of Maryland 

Fred Mueller, Pearson Learning Technologies Group 

Gary Natriello, Teachers College, Columbia University 

Zoran Popovic, University of Washington 

Seth Reichlin, Pearson 

Erin Reilly, University of Southern California 
Steve Ritter, Carnegie Learning 
Dan Schwartz, Stanford University 
David Shaffer, University of Wisconsin-Madison 
John Stamper, PSLC DataShop 

Elizabeth Tipton, Teachers College, Columbia University 
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Kurt VanLehn, Arizona State University 

Rasil Warnakulasooriya, Pearson Learning Technologies Group 
Gregory White, National Academy of Education 



Appendix B 


Summit Agenda 

Agenda for December 1-2, 2011 Summit 

Keck Center 
500 Fifth Street, NW 
Washington, DC 20001 

All locations are Keck 100 unless otherwise specified 


Day 1: Where We've Been 

8:30-9:00 am Continental Breakfast 

9:00-9:30 am Welcome 

9:30-11:00 am Demonstrations (Keck 100 and Breakout 

Rooms) 

11:00 am-12:15 pm Learning Panel 

This panel is about our ability to learn about 
learning through AET data analysis. The panel 
will focus on cognition, and social and emo- 
tional learning, as well as contextual factors. 
It will include theory building opportunities 
and the development of new learning models, 
as well as the possibilities to conduct pioneer- 
ing studies in learning and development. 
Moderator: Susan Fuhrman, Teachers 
College, Columbia University 
Panelists: 

Jere Confrey, North Carolina State 
University 

James Gee, Arizona State University 

Roy Pea, Stanford University 

12:15-12:30 pm Keynote Address: Senator Michael Bennet 

12:30-1:15 pm Lunch (Keck Cafeteria, 3rd Floor) 
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1:15-2:30 pm 

Instruction Panel 

This panel will focus on how instructional 
practices have changed because of new tech- 
nologies, as well as how they contribute to 
the ability to assess the effects of instruc- 
tional approaches. 

Moderator: Brian Rowan, University of 
Michigan 

Panelists: 

Sasha Barab, Arizona State University 
Arthur Graesser, University of Memphis 
David Pritchard, MIT 

2:30-2:45 pm 

Break 

2:45-4:00 pm 

Assessment Panel 

This panel will focus on the immediate feed- 
back on student progress allowed by these 
technologies and the possibilities for tailor- 
ing instruction as a result. 

Moderator: James Gee 

Panelists: 

Robert Mislevy, University of Maryland, 
College Park 

David Shaffer, University of 
Wisconsin-Madison 

Valerie Shute, Florida State University 

4:00-5:15 pm 

Concerns Panel 

This panel will focus on how to best address 
privacy and proprietary concerns, ensure 
quality control, and ensure theoretically 
sound analyses. 

Moderator: Brian Rowan 

Panelists: 

George Alter, Inter-University 

Consortium for Political and Social 
Research, University of Michigan 
Gary Natriello, Teachers College, 
Columbia University 

Lauren Resnick, University of Pittsburgh 
John Stamper, Carnegie Mellon 

University and Pittsburgh Science of 
Learning Center DataShop 


APPENDIX B 


39 


5:15 pm 

Reception (Keck Atrium, 3rd Floor) 

8:30-9:00 am 

Day 2: Where We're Going 

Continental Breakfast 

9:00-10:15 am 

Institutional Responses and Innovation 
Panel 

This panel focuses on how AETs influence 
institutions and on providing data-based feed- 
back to schools and other learning settings. 

Moderator: Susan Fuhrman 

Panelists: 

Richard Halverson, University of 
Wisconsin-Madison 

Ken Koedinger, Carnegie Mellon 
University 

Marcia Linn, University of California, 
Berkeley 

10:15-11:30 am 

Developing Infrastructure 

This panel includes the roles for public and 
private enterprise in building AET data analy- 
sis as a field. It will also focus on roles for a 
variety of stakeholders, including researchers, 
instructors, developers, and end users. 
Moderator: James Gee 

Panelists: 

John Behrens, CISCO 

Ed Dieterle, Bill & Melinda Gates 
Foundation 

Carl Wieman, Office of Science and 
Technology Policy, Executive Office 
of the President 


11:30 am-12:00 pm Closing 
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